Skip to content

ENH: new PyDarshan CLI tools for job/file stats for many logs #1016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 32 commits into from
Apr 30, 2025
Merged

ENH: new PyDarshan CLI tools for job/file stats for many logs #1016

merged 32 commits into from
Apr 30, 2025

Conversation

@shanedsnyder
Copy link

@shanedsnyder shanedsnyder commented Mar 10, 2025 •
edited
Loading

Based on original PR from @Yanlilyu in #954.

Key enhancements are as follows:

  • lots of new statistics/metadata printed
  • support for pretty printing via Rich package (default) or output in CSV format for subsequent analysis
  • support for PyDarshan name filtering from ENH: PyDarshan Report changes to enable name filtering #1017
  • parallelization using ProcessPoolExecutor
  • optional manifest file to support massive amounts of log file input (to avoid command line limits)
  • in-depth testing to exercise various options as part of our CI
  • updated docs to describe usage of these 2 new tools

See docs/usage.rst for details on command line tool options.

@shanedsnyder shanedsnyder changed the base branch from main to snyder/pydarshan-name-filters April 27, 2025 18:06
@shanedsnyder shanedsnyder reopened this Apr 27, 2025
@shanedsnyder shanedsnyder force-pushed the pydarshan-file-based-sorting branch from 16501ab to 3d5d197 Compare April 28, 2025 22:16
@shanedsnyder shanedsnyder reopened this Apr 28, 2025
@shanedsnyder shanedsnyder added this to the 3.4.7 milestone Apr 29, 2025
@shanedsnyder shanedsnyder force-pushed the pydarshan-file-based-sorting branch from 9ab0d49 to 05157b0 Compare April 29, 2025 19:10
@shanedsnyder shanedsnyder changed the title [WIP] New PyDarshan CLI tools for job/file stats for many logs New PyDarshan CLI tools for job/file stats for many logs Apr 29, 2025
@shanedsnyder shanedsnyder changed the title New PyDarshan CLI tools for job/file stats for many logs ENH: new PyDarshan CLI tools for job/file stats for many logs Apr 29, 2025
@shanedsnyder
Copy link
Author

shanedsnyder commented Apr 29, 2025

This is based on snyder/pydarshan-name-filters for simplicity now, but once that is merged I'll rebase on main.

@shanedsnyder
Copy link
Author

shanedsnyder commented Apr 29, 2025

Example job_stat output:
image

@shanedsnyder
Copy link
Author

shanedsnyder commented Apr 29, 2025

Example file_stats output:
image

carns
carns previously approved these changes Apr 30, 2025
View reviewed changes
Shane Snyder added 17 commits April 29, 2025 21:08
add exe and total_files columns to job_stats
44c7a30
simplify order_by column names
a2f973b
add job_stats pretty printing with rich
2250846
job_stats tweaks
8d0d341
lots of revamping of file_stats
fcfa483
add rich as pydarshan dependency
999afe6
add support for pydarshan record name filters
fc8ea34
parallelize job_stats using ProcessPoolExecutor
aa4c3e1
add partial_flag and uid to job_stats
36724db
add optional manifest file for large inputs
2d59b55
updates to file_stats
34a3a93 
- use ProcessPoolExecutor to parallelize
- allow for a input log file manifest
updated pytests for job_stats/file_stats
708f0d3
skip new dlio_logs for some test cases
d71db07
add more in-depth testing for job_stats/file_stats
3d2df38
updated pydarshan docs
608157d
skip tests when no logs repo
c5cf866
small bug fix in job_stats rich print
99634c0
@shanedsnyder shanedsnyder force-pushed the pydarshan-file-based-sorting branch from 8066582 to 99634c0 Compare April 30, 2025 02:09
@shanedsnyder shanedsnyder changed the base branch from snyder/pydarshan-name-filters to main April 30, 2025 02:10
@shanedsnyder shanedsnyder dismissed carns’s stale review April 30, 2025 02:10

The base branch was changed.

@shanedsnyder shanedsnyder reopened this Apr 30, 2025
@shanedsnyder shanedsnyder merged commit 0ca6e9e into main Apr 30, 2025
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@carns carns carns left review comments

Assignees

No one assigned

Labels

pydarshan

Projects

None yet

Milestone

3.4.7

Development

Successfully merging this pull request may close these issues.

4 participants

@shanedsnyder @carns @Yanlilyu